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FOREWORD 


The  Surveillance  Systems  Project  has  as  its  objective  the  development  of  scientific  reseai >.n 
data  bearing  on  the  extraction  of  information  from  imagery  and  the  products  of  other  sensors,  and 
the  efficient  storage,  retrieval,  and  transmission  of  information  within  an  advanced  computerized 
image  interpretation  facility.  Research  results  are  used  in  future  systems  design  and  in  the  de¬ 
velopment  of  enhanced  techniques  and  procedures  for  all  phases  of  the  image  interpretation  process 
within  the  data  reduction  facility. 

The  MAN  COMPUTER  FUNCTIONS  Task  is  one  of  four  research  Tasks  established  in  the 
Support  Systems  Research  Division  of  the  Behavioral  Science  Research  Laboratory  to  concentrate  on 
operational  segments  of  the  surveillance  system.  One  major  effort  of  the  task  is  devoted  to  the 
transfer  and  control  of  intra-system  information.  The  objective  is  to  develop  techniques  whereby 
the  computer  can  increase  the  effectiveness  of  interpretation  by  supplementing  the  decision  proc¬ 
esses  of  the  interpreter,  performing  his  routine  calculations,  and  evaluating  the  accuracy  and  com¬ 
pleteness  of  his  interpretations.  The  present  study  dealt  with  the  utility  of  feedback  presented 
under  simulated  computerized  conditions  in  improving  the  performance  of  interpreters  in  judging 
the  value  of  their  own  reports. 

The  entire  research  program  is  responsive  to  objectives  of  RDT&E  Project  2J620901A721 , 
"Surveillance  Systems:  Ground  Surveillance  and  Target  Acquisition  Interpreter  Techniques". 

FY  1966  Work  Program. 


J.  E.  UHLANER,  Dirept^r 
U.  S.  Army  Behavior6l  Science 
Research  Laboratory 


IMPACT  OF  FEEDBACK  ON  ACCURACY  OF  CONFIDENCE  LEVELS 
ASSIGNED  BY  INTERPRETERS 


BRIEF 


Requirement: 

BESRL  research  has  indicated  that  interpreters  tend  to  have  more  confidence  in  the  accuracy  of  the 
information  they  extract  from  imagery  than  is  warranted.  An  effective  method  of  improving  the  accuracy 
with  which  interpreters  evaluate  their  identifications  must  be  found  before  their  evaluations  can  be  used 
to  full  advantage  operationally.  The  present  study  explored  the  effect  on  subsequent  performance  of 
giving  interpreters  know  ledge -otresults  practice  in  rating  the  accuracy  of  their  identifications. 


Procedure: 

A  different  technique  of  providing  feedback  was  employed  with  each  of  three  experimental  groups  of 
interpreters.  A  control  group  received  no  feedback.  The  15  interpreters  in  each  group  reported  on  three 
sets  of  imagery,  one  set  on  each  of  three  successive  days,  the  same  order  of  presentation  being  followed 
in  all  groups.  In  the  three  experimental  groups,  feedback  based  on  the  previous  day’s  performance  was 
provided  at  the  start  of  the  sessions  on  the  second  and  third  days.  In  Technique  A,  the  interpreter  was 
given  a  summary  sheet  containing  a  distribution  of  his  previous  confidence  ratings  and  stating  whether 
his  ratings  were  overestimates  or  underestimates  of  the  accuracy  of  his  identifications.  In  Technique  B, 
in  addition  to  the  summary  sheet,  the  interpreter  was  given  his  scored  answer  sheets  from  the  previous 
session,  along  with  the  imagery  he  had  interpreted.  In  Technique  C,  in  addition  to  the  information  given 
in  Technique  A,  the  interpreter  was  given  a  distribution  of  ratings  and  accuracy  scores  purported  to  have 
been  made  by  several  previous  classes  of  interpreters. 


Findings: 

Results  supported  the  previous  findings  that  interpreters  do  not  as  a  rule  make  dependable  evalua¬ 
tions  of  their  identifications.  However,  the  confidence  ratings  made  by  interpreters  whose  identifications 
were  generally  more  accurate  and  complete  were  more  precise  than  those  made  by  interpreters  in  a  low 
performance  group. 

Feedback  techniques  A  and  C,  in  which  interpreters  were  given  only  data  on  previous  rating 
performance— their  own  (A)  and  their  own  plus  that  of  other  classes  (C) — resulted  in  somewhat  more 
accurate  expressions  of  confidence  than  did  Technique  B  in  which  interpreters  were  given  their  own 
corrected  reports  and  the  imagery  they  had  interpreted  in  a  previous  session.  The  confidence  ratings 
of  the  control  group,  which  had  received  no  feedback,  were  the  least  precise. 


Utilization  of  Findings: 

The  improvement  noted  with  two  feedback  methods  indicates  that  the  accuracy  of  interpreters’  confi¬ 
dence  ratings  can  be  increased  by  practice  in  applying  a  knowledge-of-results  frame  of  reference.  The 
improved  confidence  ratings,  however,  were  still  generally  inaccurate.  Evidently,  more  than  two  practice 
sess;ons  are  needed  to  enable  the  interpreter  to  reach  an  operationally  useful  level  of  accuracy  in  evalu¬ 
ating  the  information  he  provides. 
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IMPACT  OF  FEEDBACK  ON  ACCURACY  OF  CONFIDENCE  LEVELS 
ASSIGNED  BY  INTERPRETERS 


In  aerial  surveillance  systems,  image  interpreters  are  a  vital  link 
between  the  aerial  platforms  with  their  cameras  and  other  sensor  devices 
and  the  intelligence  consumer.  The  task  of  the  image  interpreter  is  to 
extract  accurate  intelligence  information  from  the  surveillance  imagery. 

Accurate  reporting  means  more  than  making  identifications  of  enemy 
targets  and  positions.  It  means  presenting  the  information  so  that  in¬ 
telligence  consumers  have  a  basis  for  Judging  the  reliability  of  the  in¬ 
formation.  To  this  end,  it  is  common  practice  ir  image  interpretation 
facilities  for  an  interpreter  to  qualify  the  information  in  his  report 
as  "positive",  "probable",  or  "possible".  In  most  military  situations, 
in  fact,  the  full  intelligence  potential  of  an  image  is  not  exploited 
unless  the  "possibles"  and  "probables"  as  well  as  the  "positives"  are 
reported.  The  image  interpreter's  confidence  in  his  identifications  can 
then  be  weighed  by  intelligence  consumers  who  utilize  the  reported  in¬ 
formation  in  making  decisions. 

The  qualitative  categories  serve  to  indicate  the  interpreter's  Judg¬ 
ment  only  in  a  very  general  way.  Too,  the  words  themselves  are  somewhat 
ambiguous.  They  may  have  different  connotations  for  different  image  in¬ 
terpreters --and  for  different  intelligence  users.  To  counter  the  like¬ 
lihood  of  ambiguity,  BESRL  has  introduced  and  used  in  all  recent  image 
interpretation  research  a  quantitative  scale  ranging  from  zero  to  100. 

It  was  felt  that  using  a  quantitative  scale  to  indicate  degree  of  confi¬ 
dence  would  reduce  ambiguity,  increase  the  range  of  confidence  values 
that  could  be  expressed,  and  allow  more  rapid  handling  of  confidence  in¬ 
formation  by  automatic  data  processing  equipment. 

Subsequent  studies,  however,  showed  that  image  interpreters  fre¬ 
quently  tend  to  have  more  confidence  in  their  identifications  than  is 
warranted  by  the  accuracy  of  the  identifications.  The  accuracy  rate  of 
some  interpreters  for  identifications  about  which  they  expressed  high 
confidence  has  been  found  on  some  BESRL  performance  measures  to  be  less 
than  50  percent^.  An  effective  method  of  improving  the  accuracy  with 
which  confidence  ratings  are  made  must  be  found  before  such  ratings  can 
be  used  more  fully  operationally.  The  present  study  explored  the  effect 
on  subsequent  Judgment  performance  of  giving  interpreters  practice  ses¬ 
sions  designed  to  improve  their  accuracy  in  assessing  their  confidence 
in  their  own  reports. 
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adacca,  R,  Martinek,  H. ,  and  Schwartz,  A.  Image  Interpretation  Task — 
Status  Report.  BESRL  Technical  Research  Report  1129,  June  1962. 


OBJECTIVES  OF  THE  STUDY 


The  primary  objective  of  the  present  study  was  to  determine  whether 
practice  in  making  confidence  ratings  can  Improve  the  accuracy  of  the 
ratings*  The  method  consisted  of  providing  the  interpreter  with  feedback 
and  then  comparing  his  stated  confidence  levels  with  the  scored  accuracy 
of  his  identifications.  Three  different  techniques  of  providing  feed¬ 
back  were  tried  out. 

A  second  abjective  was  to  determine  whether  practice  in  utilizing 
feedback  affects  the  accuracy  with  which  the  interpreters  identify 
targets  or  the  completeness  of  their  interpretation  of  an  image.  It  was 
conceivable  that  emphasis  on  making  accurate  confidence  ratings  would 
influence  the  number  of  correct  and  incorrect  target  identifications  made 
by  interpreters. 


EXPERIMENTAL  PROCEDURES 


Research  Design 

Three  experimental  groups  and  one  control  group,  each  containing  15 
subjects,  were  formed.  A  different  technique  of  providing  feedback  was 
employed  with  each  experimental  group.  All  interpreter  subjects  examined 
three  sets  of  performance  measures  in  the  same  order,  one  set  on  each  of 
three  successive  days.  Confidence  feedback  based  on  the  previous  day's 
performance  was  provided  the  interpreters  at  the  beginning  of  the  second 
and  third  sessions.  Subjects  in  the  control  group  received  no  feedback. 


Perfofmonce  Measures 

The  performance  measures  consisted  of  three  sets  of  conventional 
black  and  white  photographs  typical  of  the  operational  imagery  which  con¬ 
fronts  the  image  interpreter.  As  in  the  operational  situation,  inter¬ 
preters  were  provided  with  maps,  sortie  plot  overlays,  and  standard  refer¬ 
ences  and  photo  keys.  They  were  also  given  situation  sheets  showing  the 
number  of  photographs  in  the  performance  measures,  the  scale  of  the  photos, 
the  intelligence  information  requested,  and  the  battlefield  situation  at 
the  time  the  photos  were  obtained.  The  situation  sheets  were  read  aloud 
to  the  interpreters  before  they  began  to  examine  the  photographs  (See 
sample  situation  sheet  in  Appendix  B). 

The  interpreters  were  asked  to  detect  and  identify  objects  of  mili¬ 
tary  significance  such  as  wheeled  vehicles,  artillery,  armor,  end  forti¬ 
fications.  They  marked  directly  on  the  photographs  the  objects  they 
located  and  then  recorded  identifications  of  the  objects  on  special 
answer  sheets,  using  only  the  descriptive  terminology  provided  in  the 
Target  List  (Appendix  B). 
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Performance  on  Set  1  (T-4,  T-14,  and  T-22)*,/vas  used  to  match  the 
subjects  in  the  experimental  groups  and  to  provide  confidence  feedback. 
Performance  on  Set  2  (T-8  and  T-10)  was  used  to  measure  the  effect  of 
the  feedback  and  to  provide  confidence  estimates  for  feedback  immediately 
before  administration  of  the  third  set.  Set  3  (T-6  and  T-3)  was  used  to 
measure  the  effect  of  previous  feedback. 


Subjects 

Sixty  image  interpreter  trainees  about  to  graduate  from  the  U.  S. 
Army  Intelligence  School  at  Fort  Holabird,  Maryland  were  the  subjects. 
They  were  divided  into  four  groups  of  15  subjects  each,  matched  on  the 
basis  of  their  performance  in  assigning  confidence  ratings  to  identifi¬ 
cations  made  on  the  three  performance  measures  of  Set  1. 


Confidence  Judgments 

Each  interpreter  was  asked  to  state  the  degree  of  confidence  he  felt 
in  each  target  identification  he  made,  using  a  quantitative  scale  ranging 
from  zero  to  100$.  Instructions  for  using  the  scale  specified  that  100$ 
of  the  identifications  to  which  the  interpreter  assigned  a  confidence 
rating  of  100  should  be  correct,  80$  of  the  identifications  with  a  con¬ 
fidence  rating  of  80  should  be  correct,  and  so  forth.  The  interpreter 
was  thus  asked  to  rate  directly  the  probability  that  a  given  identifica¬ 
tion  was  correct.  Interpreters  were  cautioned  not  to  over-  or  under¬ 
estimate  their  confidence  in  an  identification  (See  Appendix  A  for  com¬ 
plete  instructions  on  recording  confidence). 


Feedback  Techniques 

The  basic  feedback  principle  employed  was  to  present  an  image  in¬ 
terpreter  with  the  accuracy  rate  he  had  achieved  in  identifying  objects 
in  the  imagery  for  each  level  of  confidence,  along  with  an  indication  of 
the  amount  of  his  over-  or  underconf idence.  Presumably,  if  an  image  in¬ 
terpreter  is  shown  through  feedback  that  he  is  consistently  overconfident 
in  his  ratings,  he  will  revise  his  Judgmental  processes  and  m^ke  more 
realistic  confidence  ratings. 

Three  feedback  techniques  were  employed.  In  Technique  A,  the  feed¬ 
back  consisted  of  presenting  each  subject  with  a  s urinary  sheet  contain¬ 
ing  a  distribution  of  his  confidence  ratings,  the  percentage  of  correctly 
identified  targets  for  each  confidence  interval,  and  an  indication  as  to 
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Designations  refer  to  performance  measures  in  the  BESRL  imagery 
library. 


whether  his  ratings  were  overestimates  or  underestimates  (See  sample 
summary  sheet  In  Appendix  B).  In  Technique  B,  each  subject  was  presented 
with  the  scored  answer  sheets  and  Imagery  from  his  previous  performance 
measure  In  addition  to  the  summary  sheet  described  above.  The  subject 
was  thereby  able  to  review  his  Identifications  and  determine  where  he  had 
made  his  errors  as  well  as  to  study  how  accurately  he  had  assigned  his 
confidence  values.  Technique  C  employed  group -oriented  feedback.  Each 
subject  was  given  the  summary  sheet  described  above.  In  addition,  he  was 
given  a  distribution  of  confidence  ratings  and  accuracy  rates  purported 
to  have  been  made  by  several  previous  classes  of  Image  Interpreters 
(Appendix  B).  These  figure  showed  that  good  agreement  between  confi¬ 
dence  ratings  and  accuracy  rates  had  been  achieved  In  previous  classes, 
but  that  on  the  average  the  Interpreters  had  been  somewhat  overconfident. 
If  Festinger's  theory  of  cognitive  dissonance^" applies  here,  the  subject 
would  presumably  experience  dissonance  due  to  his  deviation  from  the 
group  norm.  His  attempt  to  reduce  this  dissonance  would  Influence  the 
cognitive  processes  Involved  In  assigning  confidence  ratings  and  he  would 
assign  more  realistic  assessments. 


Vcriables 

The  effects  of  the  feedback  techniques  were  determined  for  the  de¬ 
pendent  variables  listed  below.  Values  for  these  variables  were  summed 
across  each  subject's  responses  to  each  set  of  performance  measures. 

Confidence  Inaccuracy  Score.  A  soore  expressing  the  degree  of  In¬ 
accuracy  of  the  confidence  ratings  made  by  the  Interpreter,  using  the 
formula  suggested  by  Adams  and  Adams*^ 


n 


1 


where  Is  the  actual  percentage  of  correct  identifications  made  at 
stated  confidence  level  P^,  and  n^  is  the  number  of  ratings  made  at  con¬ 
fidence  level  P1«  For  this  variable,  larger  scores  indicate  less  accuracy. 

Completeness  Score.  The  number  of  right  Identifications  divided  by 
the  total  number  of  targets  In  the  Imagery. 


■^"Fe stinger,  Leon.  Theory  of  cognitive  dissonance.  Evanston,  Illinois: 
Row.  1957.  — —  —  —  •  '  - . — 

Adams,  P.  A.  and  Adams,  -T.  K.  Realism  of  confidence  Judgments. 
Psychological  Review.  68,  33-35>  1961. 
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Accuracy  Score.  Tbe  number  of  right  identifications  divided  by  the 
total  number  of  identifications  made  by  the  interpreter. 


ANALYSIS  AND  RESULTS 
Overall  Accuracy  of  Confidence  Ratings 


Based  on  the  identifications  made  by  all  60  subjects  to  the  Set  1 
performance  measures ,  the  percentages  of  correct  identifications  were 
plotted  by  confidence  level  (Figure  1).  Since  the  confidence  scale  was 
defined  in  terms  of  probability  of  being  correct,  the  percentage  correct 
for  any  given  confidence  level  should  ideally  equal  the  confidence  level. 
This  ideal  relationship  is  shown  by  the  straight  line  in  Figure  1.  It 
is  readily  apparent  that  tbe  interpreters  in  the  sample  were  generally 
overconfident.  For  identifications  felt  to  have  low  probability  off 
being  correct,  however,  they  tended  to  be  less  confident  than  was 
warranted. 


Figure  1.  Percentages  of  correct  identifications  made  ot  various  confidence  levels 
by  all  interpreters  on  Set  1  imagery 
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To  determine  whether  accuracy  of  confidence  ratings  was  related  to 
interpreter  performance,  the  total  sample  was  divided  into  high  perform¬ 
ance  and  low  performance  groups  on  the  basis  of  total  right  and  wrong 
scores  made  on  the  performance  measures  in  Set  1.  As  shown  In  Table  1, 
the  confidence  ratings  made  by  the  Interpreters  in  the  high  performance 
group  were  generally  more  accurate  than  the  ratings  made  by  those  In  the 
low  performance  group.  Further  analysis  Indicated  that  the  low  perform¬ 
ance  group  was  more  overconfident  in  making  their  ratings. 


Table  1 

MEM  CONFIDENCE  INACCURACY  SCORES 
FOR  HIGH  AND  LOW  HSRFCRMANCE  GROUPS 


High  Group 

Low  Group 

(N  =  30) 

(N  -  30) 

Mean* 

35-35 

43.23 

Sigma 

6.62 

8.09 

§M«ani  Significi 

intly  Different  (p  <  .01). 

Effect  of  Feedback  on  Confidence  Inaccuracy  Scores 

To  determine  whether  the  feedback  treatments  had  any  effect  on  the 
accuracy  of  the  confidence  ratings,  a  two-way  analysis  of  variance 
(feedback  techniques  x  sets  of  performance  measures)  was  computed.  A 
repeated -measures  design  was  used  for  this  analysis,  since  each  subject 
was  administered  the  same  sets  of  performance  measures,  thereby  serving 
as  his  own  control.  The  analysis  of  variance  (Table  C-l  of  the  Appendix) 
produced  a  significant  F-ratio  for  both  the  feedback  technique  and  per¬ 
formance  measure  main  effects  (p  <  .05). 

As  shown  in  Table  2,  the  mean  confidence  inaccuracy  score  was  sig¬ 
nificantly  lower  for  feedback  techniques  A  and  C.  In  addition  to  the 
control  group,  the  group  using  Technique  B  in  which  the  interpreter  re¬ 
viewed  his  previous  identifications  and  examined  his  scores  was  least 
effective.  Interpreters  receiving  Technique  B  feedback  may  have  paid 
less  attention  to  the  stmxoary  sheets  than  the  A  and  C  groups  in  which 
the  sumnary  sheet  was  the  major  element  in  the  feedback.  The  group  norms 
presented  to  the  group  employing  Technique  C  may  have  served  to  highlight 
inaccuracies — mean  inaccuracy  scores  were  slightly  lower  for  Technique  c 
than  for  Technique  A,  where  only  the  summary  sheets  were  used. 

The  mean  confidence  inaccuracy  score  was  significantly  higher  for 
Set  3  performance  measures  than  for  Set  2.  Set  3  imagery  seemed  in 
general  more  difficult  to  interpret  Judging  from  the  scores  made  by 
interpreters  (Tables  3  and  4). 
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Table  2 


MEAN  CONFIDENCE  INACCURACY  SCORES* 
(N  «  15  using  each  technique) 


Performance  Measure  Set 


Group 

1 

2 

3 

Mean 
(2,  3) 

Feedback  Technique  A 

39.1 

32.0 

38.9 

35.5 

Feedback  Technique  B 

39.7 

39.9 

43.7 

41.8 

Feedback  Technique  C 

39.3 

29.8 

35.3 

32.5 

Control  Group 

39.5 

43.2 

49.4 

46.3 

MEAN 

39.4 

36.23 

41.82 

aMoans  significantly  different  among  feedback  techniques  (P  <  .05)  and  between  sets  2  and  3 

(P  <  .05).  Set  1  scores  were  not  Included  in  the  analysis  of  variance. 


Table  3 

MEAN  COMPLETENESS  SCORES* 

(N  =  13  using  each  technique) 


Performance  Measure  Set 

Group 

1 

2 

3 

Mean 
(2,  3) 

Feedback  Technique  A 

.30 

.28 

.13 

.22 

Feedback  Technique  B 

.30 

.31 

.19 

.25 

Feedback  Technique  C 

.27 

.27 

.17 

.22 

Control  Group 

.27 

.25 

.15 

.20 

MEAN 

.29 

.28 

.16 

.22 

aMeana  significantly  different  between  Sets 

2  and  3 (P 

<  .001).  Set  1 

scores  were  no 

t  included  In 

the  analysis  of  variance. 
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Table  4 


MEAN  ACCURACY  SCORES* 

(N  -  15  using  each  technique) 


Performance 

Measure 

Set 

Group 

1 

2 

3 

Mean 
(2,  3) 

Feedback  Technique  A 

.42 

.41 

CO 

CM 

• 

•  35 

Feedback  Technique  B 

•39 

.44 

.37 

.41 

Feedback  Technique  C 

.33 

.42 

.37 

.40 

Control  Group 

•39 

•35 

.28 

.37 

MEAN 

.39 

.40 

.32 

.36 

"Means  significantly  different  between  Sets  2  and  3  (P  <  .01).  Set  1  scores  were  not  Included  In 
the  analysis  of  variance. 


Effect  of  Feedback  on  Mean  Completeness  and  Accuracy  Scores 

To  determine  whether  the  feedback  treatments  had  any  effect  on 
general  interpretation  performance,  a  similar  analysis  of  variance  was 
computed  for  the  completeness  and  accuracy  scores.  The  only  significant 
F -ratios  were  for  performance  measure  sete. 


IMPLICATIONS 

The  primary  objective  of  the  study  was  to  determine  whether  practice 
directed  at  improving  confidence  ratings  would  increase  the  accuracy  of 
the  ratings.  The  method  consisted  of  providing  the  interpreter  with 
feedback  information  comparing  his  stated  confidence  in  his  identifica¬ 
tions  with  the  scored  accuracy  of  the  identifications.  The  secondary 
objective  was  to  determine  whether  the  feedback  practice  affected  target 
identification  accuracy  and  completeness. 

It  was  concluded  that  practice  in  which  the  interpreter  is  provided 
with  feedback  information  as  a  frame  of  reference  against  which  he  can 
assign  confidence  levels  to  identifications  he  is  currently  making  shows 
promise  for  improving  the  accuracy  of  confidence  ratings.  However,  more 
than  two  practice  sessions  are  necessary  for  the  interpreter  to  reach  an 
operationally  acceptable  level  of  accuracy.  How  many  feedback  sessions 
are  needed  for  the  interpreter  to  reach  the  desired  level  of  accuracy  in 
making  confidence  ratings  remains  to  be  determined. 
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APPENDIX  A.  INSTRUCTIONS  TO  SUBJECT  INTERPRETERS 


A-l.  INSTRUCTIONS  FOR  RECORDING  CONFIDENCE 


Your  task  is  to  record  how  confident  you  cure  that  your  identifica¬ 
tion  is  correct  in  the  column  labeled  "Ccnf."  You  are  to  use  a  scale 
that  runs  from  0  to  100,  where  100  indicates  that  you  are  certain  your 
identification  is  correct.  If  you  vise  this  scale  accurately,  all  of  the 
identifications  for  which  you  indicate  100%  confidence  should  be  correct; 
80%  of  the  identifications  for  which  you  indicate  QOf>  confidence  should 
be  correct;  5 Vf>  of  the  identifications  for  which  you  indicate  50%  confi¬ 
dence  should  be  correct,  and  so  forth.  You  can  use  10,  20,  30,  40,  50, 
60,  70,  80,  90,  or  100  to  indicate  your  estimate  of  the  probability  that 
you  have  made  a  correct  identification.  If  you  believe  you  can  make 
finer  Judgments,  you  may  do  so,  that  is,  you  might  want  to  use  75  or  95 
or  even  99#  depending  on  your  degree  of  confidence  that  your  identifica¬ 
tion  is  correct. 

From  previous  experiments,  we  have  found  that  an  interpreter's 
statements  of  confidence  in  his  identifications  are  very  important  in 
evaluating  the  accuracy  of  his  identifications;  so  try  to  be  as  accurate 
as  possible.  Try  not  to  overestimate  or  underestimate  your  confidence. 
After  your  identifications  have  been  scored,  your  ability  to  estimate 
your  confidence  accurately  will  be  determined  by  comparing  your  stated 
confidence  with  the  percent  of  correct  identifications  you  actually  made. 


-  13  T 


A -2.  INSTRUCTIONS  FOR  EXAMINEES 


TO  BE  READ  TO  GROUPS  A  AND  C  AFTER  T  _ HAS  BEEN  DISTRIBUTED 

BUT  BEFORE  THE  SITUATION  SHEET  HAS  BEEN  READ. 


You  will  find  an  additional  Information  sheet  in  your  packets  for 

T  _ .  Take  this  sheet  out  and  study  it  carefully.  You  will 

have  three  minutes  to  do  this.  Do  this  now.  Do  not  discuss  this 
information  with  any  other  member  of  this  class. 

AT  THE  END  CF  THREE  MINUTES,  PROCEED  WITH  THE  ADMINISTRATION  OF  T  _ 


TO  BE  READ  TO  GROUP*  B 


The  performance  measures  for  T _ ,  including  your  annotations 

and  your  scored  answer  sheet  have  been  returned.  Study  this  material 
carefully  to  determine  the  kinds  of  errors  you  made  and  to  ascertain 
what  led  you  to  make  these  errors.  Also  check  carefully  the  confidence 
ratings  which  you  used  to  express  your  confidence  in  both  the  wrong  and 
right  identifications.  In  addition,  study  carefully  the  additional  in¬ 
formation  sheet  contained  in  your  packets.  Ask  any  questions  you  have 
about  any  aspect  of  the  scoring.  You  will  be  allowed  20  minutes. 

AFTER  20  MINUTES :  For  the  next  sets  of  performance  measures,  I  want 
you  to  try  to  improve  the  accuracy  of  your  confidence  ratings.  Keep  in 
mind  the  errors  which  you  made  and  which  led  you  to  be  overconfident  in 
some  of  your  identifications  and  underconfident  in  others.  BE  MORE  CARE¬ 
FUL  AND  TRY  TO  BE  MORE  ACCURATE  IN  MAKING  THESE  RATINGS. 


TO  BE  READ  TO  GROUP  D 


You  will  not  need  any  additional  instructions.  Please  sit  quietly 
for  the  next  three  minutes.  DO  NOT  LOOK  AT  THE  IMAGERY  UNTIL  I  TELL  YOU 
TO  DO  SO. 
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APPENDIX  B.  MATERIALS  USED  IN  THE  EXPERIMENT 


B-l.  SAMPLE  SITUATION  SHEET 


PERFORMANCE  MEASURE:  T-4  IMAGE  INTERPRETATION  TASK 

CONTENTS:  Situation  sheet;  Immediate  Report  Form;  Photos  1-3  from  Mission 
R3923C;  Map  of  area  (scale  1:50,000);  T-4  List  of  Military 
Objects;  Sortie  plot  overlay;  and  Situation  overlay. 


GENERAL  SITUATION: 

You  are  a  member  of  the  photo  interpretation  team  assigned  to  the 
First  ROK  Corps  which  is  defending  the  right  flank  of  the  Eighth  US  Army 
Front  in  Korea  during  1952  and  1955*  The  action  along  the  entire  front 
has  been  limited  to  small  scale  probes. 


SPECIFIC  SITUATION: 

In  July  1953,  a  North  Korean  PGv  stated  that  he  had  tra vexed  through 
the  area  which  appears  on  photo  number  2,  and  saw  troop  activity,  vehicles, 
and  construction  in  the  valley  and  on  the  ridge. 

On  20  July  1953#  the  45th  TRS  flew  a  spot  reconnaissance  mission  of 
the  area.  Photos  1-3  have  been  plotted,  and  your  team  chief  has  annotated 
areas  of  suspected  activity  on  photo  2.  The  scale  of  all  photos  is 
1:5,700. 


REQUIREMENTS : 

Locate  and  identify  all  weapons,  vehicles,  and  fortifications  in 
areas  A,  B,  and  C  using  only  those  names  appearing  in  the  T-4  List  of 
Military  Objects . 

You  have  30  minutes  to  complete  this  report. 


PT  3925-4(R-2) 


Dec.  1961 


6l:3925-4(r-2) 
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B-2.  SAMPLE  TARGET  LIST 


T-4  LIST  CF  MILITARY  OBJECTS 


AA  (antiaircraft) 

Gun  (direct  fire  artillery,  not  AA) 
How  (howitzer) 

Mortar 

AW  (automatic  weapon,  not  part  of 
firing  trench) 

Missile  (missile  or  rockets) 


Car  (civilian  type) 

Light  truck  (3/4-ton  and  less) 

Truck  (larger  than  3/4-ton  truck, 
cargo  and  personnel) 

Trailer  truck  (trailer  and  tractor) 

Construction  W  (road  scrapers, 
rollers,  and  other  wheeled  con¬ 
struction  equipment) 

Trailer  (trailer  or  other  towed 
equipment ) 

Towed  artillery 

Tank  (any  size  tank  or  tracked 
self-propelled  gun) 

AFC  (any  armored  personnel  carrier) 

Construction  T  (tracked  construction 
equipment,  bulldozers,  cranes,  etc.) 


Firing  Trench  (trench  with  firing 
bays) 

Trench  (  coomo  trenches) 

Foxholes  (concentration  of  10  or  more) 

Caves  (concentration  of  10  or  more) 

Bldg  (building,  hut,  tent,  etc.) 

OP  (observation  post) 

Wire  (any  tactical  wire) 

Mines  (any  minefields) 

AT  (obstacle  which  was  constructed 
only  as  an  anti-tank  obstacle ) 

Pill  boxes 

Electronic  (radio,  radar,  etc.) 
Airfield 


FT  3925 -4 (R -2) 


Dec.  I96I 
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B-3*  SAMPLE  ADDITIONAL  INFORMATION  SHEET 
Feedback  Presented  to  Groups  A  and  B 


Man  No.  _ 

From  previous  experience,  it  has  been  determined  that  an  Interpreter's 
statement  of  confidence  Is  very  Important  to  a  commander  In  evaluating  the 
accuracy  of  the  identifications.  Performance  measures  T-14,  T-4,  and  T-22 
have  been  scored  analyzed.  The  accuracy  of  your  confidence  statements 
for  these  measures  has  been  determined  by  comparing  your  stated  confidence  in 
your  identifications  of  all  targets  with  the  percentage  of  targets  correctly 
identified.  The  table  below  sunmarizes  the  accuracy  of  your  confidence  state¬ 
ments  for  these  three  measures.  This  table  indicates  how  accurate  your  con¬ 
fidence  statements  were,  whether  they  were  overestimates  or  underestimates, 
i.e.,  whether  you  were  overconfident  or  lacked  confidence  In  your  identifica¬ 
tions.  Read  this  table  carefully  before  starting  your  next  performance 
measure.  Try  to  Improve  your  confidence  estimations— 90^  of  the  identifica¬ 
tions  you  make  with  a  confidence  of  "90"  should  be  correct,  80 36  of  the 
identifications  you  make  with  a  confidence  of  "80"  should  be  correct,  and 
so  on. 


ACCURACY  OF  CONFIDENCE  STATEMENTS  FOR  PERFORMANCE 

MEASURES  T-14,  T-4,  T-22 

Targets 

$  Confidence  Statements 

Confidence 

No.  Targets 

Correctly 

Over 

Under 

Intervals 

Identified 

Identified 

Estimates 

Estimates 

81-100 

9 

55 

55 

61-80 

7 

14 

56 

60  and  below 

13 

6 

24 
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b-4.  sample  confidence  estimate  summary  sheet 


Additional  Feedback  Presented  to  Group  C  Only 


.  ACCURACY  CF  CONFIDENCE  ESTIMATES  FOR  PERFORMANCE  MEASURES,  T-8  AND  T-10 

Accuracy  of  Prior  Classes 

Your  Performance 

Confidence 

Estimate 

1>  Targets 
Correctly 
Identif  led 

Estimate 

Confidence 

Estimate 

$  Targets 
Correctly 
Identified 

Estimate 

Over 

io  Under 

Over 

%  Under 

100 

85 

15 

100 

100 

0 

0 

80-99 

76 

14 

80-99 

67 

23 

60-79 

58 

12 

60-79 

100 

30 

40-59 

38 

12 

40-59 

13 

37 

Under 

40 

25 

5 

Under 

40 

25 

5 

Tbe  above  table  presents  accuracy  of  confidence  estimate  data  from  several 
previous  classes  on  Performance  Measures  T-8  and  -10.  Note  that  tbe  amount  of 
overconfidence  for  these  tests  is  less  than  that  for  T-4,  -14,  and  -22,  but  there 
is  still  roan  for  improvement. 

Your  performance  on  T-8  and  -1C  shows  that  you  are  still  a  little  OVERCONFI- 
DEOT  in  your  expressions  of  confidence  in  your  identifications,  with  the  exception 
of  estimates  in  the  60  to  79  range,  where  you  appear  to  be  a  little  cautious. 

Using  this  information  as  a  guide,  on  the  next  series  of  Performance  Measures, 
try  to  adjust  your  confidence  estimates  so  that  they  are  more  accurate.  Check  each 
identification  carefully  before  expressing  your  confidence  in  the  identification. 
Remember,  a  target  identif iea  with  a  confidence  estimate  of  100  should  be  correctly 
identified  100  percent  of  the  times,  while  a  target  identified  with  a  confidence 
estimate  of  50  should  be  correctly  identified  ONLY  50  percent  of  the  times. 
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APHSNDIX  C.  ANALYSIS  CF  VARIANCE  TABIES 


Table  C-l 

MEAN  INACCURACY  INDEX 


Source 

df 

MS 

F 

P 

Feedback  Methods  (M) 

3 

1154.3 

3.78 

.05 

Subjects  ■  e^ 

56 

305.1 

Test  Series  (T) 

1 

946.4 

4.90 

.05 

M  x  T 

3 

12.9 

.07 

NS 

Subjects  x  T  ■  e^ 

56 

193.1 

Table  C-2 

ACCURACY  CF  IDENTIFICATION 

Source 

df 

MS 

F 

P 

Feedback  Methods  (M) 

3 

522.8 

.96 

NS 

Subjects  =  e^ 

56 

542.2 

Test  Series  (T) 

1 

I833.O 

5.99 

.05 

M  x  T 

3 

77.5 

.25 

Subjects  x  T  =  ea 

56 

305.8 
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Table  C-3 

CCMPI£TENES3  OF  IDENTIFICATION 


Source 

df 

MS 

F 

P 

Feedback  Methods  (M) 

3 

133.1 

.85 

NS 

Subjects  =  e^ 

56 

157.3 

Test  Series  (T) 

1 

4013.6 

30.2 

.001 

M  x  T 

3 

32.9 

•25 

NS 

Subjects  x  T  =  e^ 

56 

132.8 

-  20 


Unclassified 


_ Security  Classification _ 

“  DOCUMENT  CONTROL  DATA  •  R&D 

(Security  elaaaltlcatlon  of  tltla.  body  ot  abstract  and  Inditing  annotation  mutt  ba  an  f  rad  *vt>#n  th a  ovaralt  raport  la  claaalhad) 


)  REPORT  TITLE 

Impact  of  Feedback  on  Accuracy  of  Confidence  Levels  Assigned  by  Interpreters 

4  DESCRIPTIVE  NOTES  (Typa  ot  nport  and  inctuaiva  do  too) 

I  4UTHORCS;  (Laat  nama,  tint  namo.  Initial) 

Thomas,  James  A.  and  Sadacca,  Robert 


•  REPORT  DATE 

_ June  19^7 _ 

%a  CONTRACT  OR  GRANT  NO. 

6  PROJECT  NO 

DA  R&D  Proj.  No.  2J620901A721 

c. 

Man-Computer  Functions 
d  b-00 


7  a  TOTAL  NO  OF  PAGFS 

76  NO  or  REF* 

29 

3 

t«  ORIGINATOR'S  REPORT  NUMBERS; 


Technical  Research  Note  187 


•  b.  OTHER  REPORT  NO(S)  (A  ny  othar  numbara  that  may  ba  aaalfaad 
thla  nport ) 


2  a  REPORT  SECURITY  CLASSIFICATION 

Unclassified 

26  CROUP 


1  ORIGINATING  ACTIVITY  < Corporal a  author; 

U.  S.  Army  Behavioral  Science  Research  Laboratory, 
OCRD,  Washington,  D.  C. 


Unclassified 

Security  Classification 


14 

KEY  WORDS 

♦Image  systems — work  methods 
Image  interpretation 
Intra-system  information — transfer 
Computer-based  image  interpretation  systems 
Laboratory  facilities 
♦Feedback  techniques 

♦Image  interpretation  performance  measures 


INSTRUCTIONS 


1.  ORIGINATING  ACTIVITY:  Enter  the  name  and  address 
of  the  contractor,  subcontractor,  grantee.  Department  of  De¬ 
fense  activity  or  other  organization  ( corporate  author)  issuing 
the  report. 

2a.  REPORT  SECUWTY  CLASSIFICATION:  Enter  the  over¬ 
all  security  classification  of  the  report.  Indicate  whether 
**  Restricted  Data"  is  included.  Marking  is  to  be  in  accord¬ 
ance  with  appropriate  security  regulations. 

2b.  GROUP:  Automatic  downgrading  is  specified  in  DoD  Di¬ 
rective  5200.  10  and  Armed  Forces  Industrial  Manual.  Enter 
the  group  number  Also,  when  applicable,  show  that  optional 
markings  have  been  used  for  Group  3  and  Group  4  as  author¬ 
ized 

3.  REPORT  TITLE  Enter  the  complete  report  title  in  all 
capital  letters.  Titles  in  all  cases  should  be  unclassified. 

If  a  meaningful  title  cannot  be  selected  without  classifica¬ 
tion,  show  title  classification  in  all  capitals  in  parenthesis 
immediately  following  the  title. 

4.  DESCRIPTIVE  NOTES:  If  appropriate,  enter  the  type  of 
report,  e.g. ,  interim,  progress,  summary,  annual,  or  final. 

Give  the  inclusive  dates  when  a  specific  reporting  period  is 
covered. 

5.  AUTHOR(S):  Enter  the  name(s)  of  author(s)  as  shown  on 
or  in  the  report.  Enter  last  name,  first  name,  middle  initial. 

If  military,  show  rank  and  branch  of  service.  The  name  of 
the  principal  author  is  an  absolute  minimum  requirement. 

6.  REPORT  DATE  Enter  the  date  of  the  report  as  day, 
month,  year,  or  month,  year.  If  more  than  one  date  appears 
on  the  report,  use  date  of  publication. 

7 a.  TOTAL  NUMBER  OF  PAGES:  The  total  page  count 
should  follow  normal  pagination  procedures,  i.e.,  enter  the 
number  of  pages  containing  information. 

7b.  NUMBER  OF  REFERENCES  Enter  the  total  number  of 
references  cited  in  the  report. 

8a.  CON  RACT  OR  GRANT  NUMRER:  If  appropriate,  enter 
the  applicable  number  of  the  contract  or  grant  under  which 
the  report  was  writter 

8b,  8c,  &  8d  PROJECT  NUMBER:  Enter  the  appropriate 
military  department  identification,  such  as  project  number, 
subproject  number,  system  numbers,  task  number,  etc. 

9a.  ORIGINATOR’S  REPORT  NUMBER(S):  Enter  the  offi- 
cial  report  number  by  which  the  document  will  be  identified 
and  controlled  by  the  originating  activity.  This  number  must 
be  unique  to  this  report. 

9b  OTuER  REPORT  NUMBER(S):  If  the  report  has  been 
assigned  any  other  report  numbers  (either  by  the  onfimator 
or  bv  the  sponsor),  also  enter  this  number(s). 


10.  AVAILABILITY/LIMITATION  NOTICES:  Enter  any  lim¬ 
itations  on  further  dissemination  of  the  report,  other  than  those 
imposed  by  security  classification,  using  standard  statements 
such  as: 

(1)  "Qualified  requesters  may  obtain  copies  of  this 
report  from  DDC" 

(2)  "Foreign  announcement  and  dissemination  of  this 
report  by  DDC  is  not  authorized." 

(3)  "U.  S.  Government  agencies  may  obtain  copies  of 
this  report  directly  from  DDC.  Other  qualified  DDC 
users  shall  request  through 


(4)  "U.  S.  military  agencies  may  obtain  copies  of  this 

report  directly  from  DDC.  Other  qualified  users 
shall  request  through 


"All  distribution  of  this  report  is  controlled.  Qual¬ 
ified  DDC  users  shall  request  through 


If  the  report  has  been  furnished  to  the  Office  of  Technical 
Services,  Department  of  Commerce,  for  sale  to  the  public,  indi¬ 
cate  this  fact  and  enter  the  price,  if  known. 

11.  SUPPLEMENTARY  NOTES:  Use  for  additional  explana¬ 
tory  notes. 

12.  SPONSORING  MILITARY  ACTIVITY:  Enter  the  name  of 
the  departmental  project  office  or  laboratory  sponsoring  (pay¬ 
ing  for)  the  research  and  development.  Include  address. 

13  ABSTRACT  Enter  .in  abstract  giving  a  brief  and  factual 
summary  of  the  document  indicative  of  the  report,  even  though 
it  may  also  appear  elsewhere  in  the  body  of  the  technical  re¬ 
port.  If  additional  spare  is  required,  a  continuation  sheet 
shall  he  attached 

It  is  highly  desirable  that  the  abstract  of  classified  re 
ports  be  unclassified.  Each  paragraph  of  the  abstract  shall 
end  with  an  indication  of  the  military  security  classification 
of  the  information  in  the  paragraph,  represented  as  (TS),  (S). 
(C).  or  (U) 

There  is  no  limitation  on  the  length  of  the  abstract.  How¬ 
ever,  the  suggested  length  is  from  ISO  to  22S  words 

14  KEY  Vi  OR  DS:  Key  words  are  technic  ally  meaningful  terms 
or  short  phrases  that  characterize  a  report  and  may  be  used  as 
index  entries  for  c  ataloging  the*  report.  Key  words  must  be 
selected  so  that  no  security  classification  is  required.  Iden- 
fiers,  such  as  equipment  model  designation,  trade  name,  mill 
tary  project  code  name,  geographic  loc  at  ion,  may  be  used  as 
key  words  but  will  be  followed  b\  an  indication  >f  technical 
context  The  assignment  of  links  rules,  and  weights  is 
optional 


22 


Unclassified _  _ 

Security  Classification 


DD  Form  1473 

13.  ABSTRACT  continued 

knowledge -of -results  frame  of  reference.  Findings  suggest,  however, 
that  more  than  two  practice  sessions  are  needed  for  the  interpreter  to 
reach  an  operationally  useful  level  of  accuracy  in  evaluating  the  in¬ 
formation  he  provides. 


-  23  - 


